Method and system for forming an acoustic signal from neural timing difference data

ABSTRACT

A non-invasive system and process for converting sensory data, e.g., visual, audio, taste, smell or touch, to neural firing timing differences in a human brain and using acoustic signals to generate the neural firing time differences. Data related to neural firing time differences, the acoustic signals, and a user&#39;s response map may be stored in memory. The user&#39;s response map may be used to more accurately map the calculated neural firing time differences to the correct neural locations.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of co-pending U.S. patent applicationSer. No. 10/441,390, filed May 20, 2003, entitled “Method And System ForForming An Acoustic Signal From Neural Timing Difference Data,” which isa continuation of U.S. patent application Ser. No. 09/690,786 filed Oct.17, 2000, (now U.S. Pat. No. 6,584,357, issued Jun. 24, 2003). Thisapplication is also related to U.S. patent application Ser. No.09/690,571 filed on Oct. 17, 2000, (now U.S. Pat. No. 6,536,440, issuedMar. 25, 2003). Each of the above-referenced applications are assignedto the Assignee of the present invention, and are hereby incorporated byreference in their entireties.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a method and system for generatingsensory experiences. In particular, the present invention relates to amethod and system for forming an acoustic signal from neural timingdifference data.

2. Description of Related Art

A conventional technique for generating neural activity in the humannervous system requires surgical implants. The implants may compriseelectronic connections and wires that cause electronic impulses tointeract with some portion of the human nervous system, such as thehuman neural cortex, and thereby cause neural activity in the humanneural cortex. Researchers have successfully mapped audio sensory datato the cochlear channel, and visual data to the visual cortex.

Conventional invasive techniques have several drawbacks. First, surgicalimplants may cause patient trauma and medical complications duringand/or after surgery. Second, additional or on-going surgery may berequired, particularly if new technology is developed.

SUMMARY OF THE INVENTION

The present invention solves the foregoing drawbacks by providing anon-invasive system and process that uses acoustic signals to generatesensory data, e.g., visual, audio, taste, smell or touch, within/ontothe human neural cortex. The system forms acoustic signals from neuraltiming difference data.

One advantage of the system is its adaptability to each individual user.Human brains have some similarities, but they may vary in size, shape,number of convolutions, etc. The present system comprises componentsthat may be calibrated and a library of acoustic signals that may becustomized for each individual user. The system is advantageouslyconfigured to allow vision-impaired and/or hearing-impaired users toexperience at least some visual and/or auditory sensations.

Another advantage of the system is that no invasive surgery is needed toassist a person, such as a blind or deaf person, to experience live orrecorded images or sounds.

One embodiment of the system comprises a primary transducer array and asecondary transducer array. The primary transducer array acts as acoherent or nearly-coherent signal source. The secondary transducerarray acts as a controllable, acoustic diffraction pattern that shapes,focuses and modulates energy from the primary transducer onto the neuralcortex in a desired pattern. The secondary transducer emits acousticenergy that may be shifted in phase and amplitude relative to theprimary array emissions.

The projected, ultrasonic sensory pattern of energy is configured suchthat each portion of the pattern projected into the neural cortex may beindividually pulsed at low frequencies. The system produces lowfrequency pulsing by controlling the phase differences between theemitted energy of the primary and secondary transducer array elements.The ultrasonic signal pulsed at low frequencies affects the neuralfiring timing in the cortex. Even though a person may be blind or havehis or her eyes closed, the person's visual cortex neurons are stillfiring. Changes in the neural firing timing induce various sensoryexperiences, depending on the altered firing time and the location ofthe neuron in the cortex. The mapping of some sensory areas of thecortex is known and used in current surgically invasive techniques. Thepresent system induces recognizable sensory experiences by applyingultrasonic energy pulsed at low frequency in one or more selectedpatterns on one or more selected locations of the cortex.

One aspect of the invention relates to a method of storing data relatedto acoustic signals configured to alter neural firing times in a brain.The method comprises non-invasively projecting a first acoustic signalinto the brain. The first acoustic signal affects a neural firing timeat a first neural location in the brain. The method stores a usersensory response and data related to the first acoustic signal in amemory. The method non-invasively projects a second acoustic signal intothe brain, and stores a user sensory response and data related to thesecond acoustic signal in the memory.

Another aspect of the invention relates to a method of customizing alibrary of data related to acoustic signals configured to alter neuralfiring times in a brain. The method comprises retrieving data related toa first acoustic signal from a memory; projecting a first acousticsignal into the brain using the data related to a first acoustic signal,the first acoustic signal affecting a neural firing time at a firstneural location in the brain; storing a user sensory response with thedata related to the first acoustic signal in the memory; retrieving datarelated to a second acoustic signal from the memory; projecting a secondacoustic signal into the brain using the data related to the secondacoustic signal; and storing a user sensory response with the datarelated to the second acoustic signal in the memory.

Another aspect of the invention relates to a system of storing datarelated to acoustic signals configured to alter neural firing times in abrain. The system comprises a transducer system configured tonon-invasively project a first acoustic signal and a second acousticsignal into the brain, the first and second acoustic signal affectingone or more neural firing times at one or more neural locations in thebrain; a signal generator coupled to the transducer system; and a memorycoupled to the signal generator. The memory is configured to store: datarelated to the first and second acoustic signals; and user sensoryresponses produced by the first and second acoustic signals. The signalgenerator is configured to select data in the memory related to signalsconfigured to generate the neural firing time differences in the brain,the transducer system is configured to apply the signals to generate theneural firing time differences in the brain.

The present invention will be more fully understood upon considerationof the detailed description below, taken together with the accompanyingdrawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates one embodiment of a system for generating sensorydata onto a human neural cortex.

FIG. 2 illustrates a method for calibrating the system of FIG. 1 whichgenerates sensory data onto a human neural cortex.

FIG. 3 illustrates a method of generating sensory data onto a humanneural cortex with the system of FIG. 1.

Use of the same reference symbols in different figures indicates similaror identical items.

DETAILED DESCRIPTION

FIG. 1 illustrates one embodiment of a system1 120 for generatingsensory data onto a human neural cortex. The system 120 comprises areceiving module 110, a processing module 101, a signal generator 102, areference signal generator 103, a transducer system 106, a first signalline 104, a second signal line 105, a memory 140 and an input/outputdevice 144. All of the components, except the memory 140 and theinput/output device 144, are described in U.S. Pat. No. 6,536,440, whichis assigned to the Assignee of the present invention, and is herebyincorporated by reference in its entirety.

One or more of the components illustrated in FIG. 1, such as thetransducer system 106, may be specially configured to generate visual,audio, taste, smell and/or touch within the human neural cortex. In oneembodiment, some or all of the components of FIG. 1 may be integrated ina light-weight, compact device that may be strapped to a user, e.g. in abackpack or belt pack.

In FIG. 1, the memory 140 is coupled to at least the signal generator102 and/or the reference signal generator 103. The memory 140 maycomprise any suitable type of memory that is preferably compact andadapted for fast memory access. The input/output device 144 is coupledto at least the memory 140. The input/output device 144 may comprise akeypad, a mouse, a display or other type of suitable input/output devicethat allows an administrator or user to calibrate the components of thesystem 120 and/or modify the data stored in the memory 140.

The memory 140 stores a library 142 of neural firing time data and/orneural firing time difference data. The system 120 uses the data in thelibrary 142 to generate an acoustic signal or pattern which alters, e.g.speeds up or slows down, one or more neural firing times of the humanbrain 100A. The patterns may affect various portions of the brain 100Asubstantially simultaneously. For example, the transducer system 106 mayuse signal phase shifts between two ultrasonic sources, such as theprimary and secondary transducer arrays 130, 132, to produce specificpulse patterns which modify the firing times of targeted neurons. In oneembodiment, the transducer system 106 produces a high frequency patternthat is pulsed at low frequencies. Altering the neural firing timescauses a user to perceive sensory experiences.

The resolution, color, accuracy and other characteristics of thegenerated sensory experiences may vary according to the type oftransducers used, the amount of neural firing time data stored in thelibrary 142, and the processing power and speed of the system 120. Forexample, high resolution may be achieved with a large amount of neuralfiring time data and transducer arrays configured to focus acousticsignals to very small areas of the brain 100A.

The neural firing time data is obtained by reversing or inverting theacts of a technique described in “Reconstruction of Natural Scenes fromEnsemble Responses in the Lateral Geniculate Nucleus” by Garrett B.Stanley et al. in the Sep. 15, 1999 issue of the Journal ofNeuroscience, which his hereby incorporated by reference in itsentirety. Stanley et al. describes a technique of reconstructingspatiotemporal natural scenes by linearly decoding ensemble responseswith the later geniculate nucleus (177 cells) of a cat. The presentmethod and system reverses Stanley's technique in order to convertsensory data to neural firing time data and use a pattern of ultrasoundsignals based on the neural firing time data to alter neural firingtimes within the brain 100A. The altered neural firing times, i.e.,neural firing time differences, generate sensory experiences for theuser.

The use of single ultrasound pulses to modify nerve excitability isdescribed in “Transient Modification of Nerve Excitability In Vitro bySingle Ultrasound Pulses” by Mihran et al. found in the Department ofElectrical and Computer Engineering, University of Colorado, 1990, paper#90-038, which is hereby incorporated by reference in its entirety.Human hearing and the action of ultrasound are described in “HumanHearing In Connection With The Action of Ultrasound In the MegahertzRange On The Aural Labyrinth” by L. R. Gavrilov in the Sov. Phys.Acoust. 26(4), July-August 1980, pages 290-292, which is herebyincorporated by reference in its entirety.

During the manufacture of the system 120, a manufacturer may configureand store data in the memory 140, as well as calibrate the components ofthe system 120. The library 142 may comprise pre-determined or testeddata related to different signals which are categorized into groups,such as signals generating visual experiences, signals generatingauditory experiences, signals generating tactile experiences, etc. Thegroups may be further sub-categorized based on the size, shape, brightor dark, color, duration, pitch, etc. of the sensory experiences.

The library 142 may be complete, partially incomplete or substantiallyempty after manufacturing. An administrator at a user site may use theinput/output device 144 to modify or add data in the library 142 basedon responses from a current user or a previous user of the system 120.

In one embodiment, there is a library of various signals that may beapplied to each neural location of the brain 100A or a part of thebrain, such as the visual cortex 100. For example, if there are 100neural locations mapped, then there may be 100 libraries of signals. Asused herein, a neural location may comprise a single neuron or a groupof neurons.

In one embodiment, there is a library of various signals for eachtransducer element in the primary and secondary transducer arrays 130,132. The transducer arrays 130, 132 may be two-dimensional orthree-dimensional arrays. A desired ultrasonic pattern in the brain 100Agenerated by the primary and secondary transducer arrays 130, 132 (e.g.phased arrays) may be calculated by adding the waves generated by eachtransducer element.

FIG. 2 illustrates a method for calibrating or configuring the system120 of FIG. 1 which generates sensory data onto a human neural cortex ofa particular user's brain 100A. In a start block 200, the administratorattaches the transducer system 106 in FIG. 1 non-invasively to a user'shead and powers on the system 120. In one embodiment, the transducersystem 106 is positioned near the back of the user's head to be closerto the visual cortex 100. The transducer system 106 may be attached andremoved by the administrator or the user.

In a block 202, the administrator causes the transducer system 106 togenerate a high frequency acoustic signal(s)/pattern pulsed at lowfrequencies into the user's brain 100A shown in FIG. 1. An initialsignal may be called a ‘test signal.’

In a block 204, the signal(s) affects, e.g. speeds up or slows down, oneor more neural firing times in the user's brain 100A, such as the visualcortex 100.

In a block 206, the user describes a sensory experience to theadministrator. For example, if the transducer system 106 is configuredto generate sensory experiences in the visual cortex, the user mayexperience a flashing light, a ramp from a bright area to a dark area,or an object at a particular location of the user's simulated visualfield. If the transducer system 106 is configured to generate sensoryexperiences in the cochlear channel, the user may experience a sound ofa particular frequency, amplitude and duration.

In a block 208, the administrator may calibrate the system 120 based onthe user's described sensory experience. For example, the administratormay calibrate the processing module 101, the signal generator 102, thereference signal generator 103 and/or the transducer system 106 based onthe user's described sensory experience. If the signal was supposed togenerate a bright white square in the top left corner of the user'ssimulated visual field, the administrator may calibrate the system 120such that the user will perceive a bright white square the next time asignal is sent. The administrator may use the input/output device 144 orsome other suitable device to calibrate the system 120.

Instead of or in addition to calibrating the system 120, theadministrator may modify the data in the library 142 stored in thememory 140 based on the user's described sensory experience. Theadministrator may also enter new data associated with the primary and/orsecondary transducer arrays 130, 132 into the library 142 with theinput/output device 144.

In a block 210, the administrator may repeat the acts in blocks 200-208a plurality of times to fill a partially incomplete library 142 and/orto achieve a level of sensory accuracy or resolution desired by theadministrator or the user. Subsequent signals may vary in frequency,amplitude, duration and location. For example, the administrator may usethe system 120 to create a map of various signals with variouscharacteristics applied to various location of the brain 100A or a partof the brain 100A that corresponds to various perceived visual images.

In one embodiment, the administrator uses the system 120 to create a‘visual field’ of perceived visual ‘pixels’ in memory 140 by testing aplurality of neural locations in the visual cortex 100. The ‘pixel’ mayvary from light to dark or from colored to non-colored. Theadministrator may use the system 120 to map several degrees of light orcolor intensity for each pixel. The resolution of the visual fielddepends on (i) the focusing capability of the transducer system 106,(ii) a number of different neural locations tested by the administrator,and (iii) a number of different neural firing time differences appliedat each neural location by the administrator slightly altering theamplitude, frequency, etc. of the test signal. Thus, the systemcomponents and/or the library 142 may be customized to each individualuser.

Data in a library 142 may be transferred from memory 140 to othermemories or to a database. Various transfer methods may be used,including wire, cable, and wireless communications systems.

FIG. 3 illustrates a method of generating sensory data onto a humanneural cortex. The system 120 may be configured to generate live orrecorded images, videos, textual pieces, sounds, audio pieces, smells,taste and tactile sensations. In a block 300, the receiving module 110of FIG. 1 receives a sensory input from a video camera or other source,such as a VCR, a DVD player, a cable TV system, an Internet connection,etc. The sensory input may be transmitted by a wire or wirelesscommunication system. For example, for a vision-impaired user, the videocamera may be strapped on or near the user's heard such that the angleof the camera changes as the user turns his or her head. Alternatively,the video camera may be configured to move according to ahand-controlled device, such as a computer game joy stick. The sensoryinput may comprise digital data or analog data. If the input data isanalog, the processing module 101 may digitize the input data.

In a block 302, the processing module 101 and/or the signal generator102 calculates neural firing time differences for mapped locations ofthe visual cortex 100 based on the sensory input.

In a block 304, the signal generator 102 selects data in the library 142that will be used by the transducer system 106 to generate signals andachieve the desired neural firing time differences in the brain 100A. Inone embodiment, the signal generator 102 selects data from the library142 related to at least one pulse shaping signal, e.g. phase shift, foreach targeted location in the visual cortex 100. For example, if thereare 900 targeted locations in the visual cortex 100, then the signalgenerator 102 selects an individual pulse shaping signal from thelibrary 142 for each of the 900 neural locations. The selected signalsmay vary in amplitude, phase, and/or duration.

In a block 306, the signal generator 102 sums the selected pulse shapingsignals into a final applied signal or pattern for the secondarytransducer array 132.

In a block 308, the reference signal generator 103 may select areference signal shaping based one or more factors, such as (1) thesize, shape and configuration of the transducer system 106, and (2) thetype of signals used by the secondary transducer array. The transducersystem 106 may comprise a variety of transducer shapes, sizes,configurations, etc. Data related to various reference signals,including reference signals to generate a planar wave, may be stored inthe library 142. The reference signals may be configured and stored by amanufacturer when the system 120 is manufactured and/or modified by anadministrator at a user site.

The reference signals generated by the primary transducer array 130 mayfocus or shape the pattern generated by the secondary transducer array132. The reference signals may vary in amplitude, phase, and/or durationfrom the signals selected by the signal generator 102.

In a block 310, the signal generator 102 applies a summed pulse-shapingsignal to the secondary transducer array 132, and the reference signalgenerator 103 applies a reference signal to the primary transducer array130. The transducer arrays 132, 130 generate a pulsed ultrasoundsignal(s) or pattern comprised of phase shifts to the brain 100A, andthe user experiences a sensory experience based on the sensory inputfrom the video camera or other input source. The generated sensoryexperience may be may not be exact, but the generated sensory experienceat least gives the user an idea of the sensory input. For example,depending on the implementation, a user using the system 120 may be ableto only ‘see’ an outline of objects in front of the video camera.

In one embodiment, the ultrasound signals or pattern may be continuous,such that the user perceives a visual image in real-time as the videocamera receives the image. In another embodiment, the ultrasound signalsor pattern may be almost continuous, such that the user perceives avisual image in almost real-time, i.e., a string of snap shots, as thevideo camera receives the image.

Various types of memories, input/output devices, caches, controllers,registers and/or processing components may be used in accordance withthe present invention. The scope of the present invention is not limitedto a particular type of memory, input/output device, cache, controller,register and/or processing component. Various embodiments of the system160 may comprise other components in addition to or instead of thecomponents shown in FIG. 2 without departing from the scope of theinvention. For example, the system 160 may comprise a sensory inputdevice, additional memories, caches, controllers, registers and/orprocessing components.

The above-described embodiments of the present invention are merelymeant to be illustrative and not limiting. It will thus be obvious tothose skilled in the art that various changes and modifications may bemade without departing from this invention in its broader aspects. Theappended claims encompass all such changes and modifications as fallwithin the true spirit and scope of this invention.

1. A method of customizing a library of data related to acoustic signalsconfigured to alter neural firing times in a brain, the methodcomprising: retrieving data related to a first acoustic signal from amemory; projecting a first acoustic signal into the brain using the datarelated to a first acoustic signal, the first acoustic signal configuredto affect a neural firing time at a first neural location in the brain;storing a user sensory response with the data related to the firstacoustic signal in the memory; retrieving data related to a secondacoustic signal from the memory; projecting a second acoustic signal inthe brain using the data related to the second acoustic signal; andstoring a user sensory response with the data related to the secondacoustic signal in the memory.
 2. The method of claim 1, wherein thefirst and second acoustic signals each comprise a pulsed signalgenerated by a primary transducer array and a secondary transducerarray, the primary transducer array generating a reference wave and thesecondary transducer array generating a diffraction pattern.
 3. Themethod of claim 1, further comprising calibrating components thatgenerate the first and second acoustic signals.
 4. The method of claim1, further comprising transferring the user sensory responses and thedata in the memory to a second memory.
 5. A non-invasive system forprojecting sensory data in a part of a human brain, the systemcomprising: a primary transducer array configured to emit acousticenergy as a coherent signal source toward the human brain; a secondarytransducer array positioned in a predetermined position relative to theprimary transducer array and the human brain; and a sensory dataprocessing system coupled to the secondary transducer array, wherein thesensory data processing system sends an acoustical pattern signal to thesecondary transducer array, the secondary transducer array producing adiffraction pattern for the emitted energy from the primary transducerarray, the diffraction pattern configured to alter neural firing timingin the brain, wherein the sensory data processing system obtains sensorydata from a data source selected from a group consisting of a videocamera, a VCR, a DVD player, a cable broadcast, a satellite broadcast,and an Internet connector.